Image processing device, image capturing device, and image processing method

ABSTRACT

According to one embodiment, an image processing device includes a hardware processor implemented by one or more processors. The hardware processor acquires a first image of a first object and a second image of a second object which is different from the first object, the images captured by an image capture processing device including an image capturing element and an optical system which images an object image on an image capturing surface of the image capturing element. The hardware processor measures a point spread function of the optical system based on the first image and the second image.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is based upon and claims the benefit of priority fromJapanese Patent Application No. 2018-222647, filed Nov. 28, 2018, theentire contents of which are incorporated herein by reference.

FIELD

Embodiments described herein relate generally to an image processingdevice, image capturing device, and an image processing method.

BACKGROUND

A technique for obtaining a distance to an object from images capturedby two cameras or a stereo camera (binocular camera) is known.Furthermore, in recent years, there is a technique proposed to obtain adistance to an object from images captured by one camera (monocularcamera).

As a method of obtaining a distance to an object from images captured byone camera at once, there is a technique of acquiring the distance usingblur information. In this method, the distance is calculated using, forexample, blur information obtained from the captured images andpreliminarily-prepared correction filters. The correction filters arepreliminarily prepared per distance using a point spread function (PSF)which is simplified.

However, such a simplified point spread function is different from apoint spread function of an optical system of a camera, and thus, when adistance is calculated using the preliminarily-prepared correctionfilters, an error may possibly occur in the calculated distance. Thus,in order to improve the accuracy of the calculated distance, correctionfilters prepared using the point spread function of the optical systemof the camera are desired instead of using the simplified point spreadfunction. However, measurement of the point spread function of theoptical system of the camera must be performed in consideration ofinfluences from environmental light and a display system, and thus, isdifficult to perform.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates an example of the structure of an image capturingdevice of an embodiment.

FIG. 2 illustrates an example of a system structure of an imageprocessing unit of the embodiment.

FIG. 3 illustrates an example of the structure of a filter of theembodiment.

FIG. 4 illustrates an example of characteristics of transmissivity ofthe filter of the embodiment.

FIG. 5 illustrates changes of light rays by a color aperture in whichthe filter of the embodiment is disposed and shapes of blur.

FIG. 6 illustrates an example of a functional structure of an imageprocessing program of the embodiment.

FIG. 7 illustrates an example of a blur function of a reference image ofthe embodiment.

FIG. 8 illustrates an example of a blur function of a target image ofthe embodiment.

FIG. 9 illustrates an example of a blur correction filter of theembodiment.

FIG. 10A illustrates a measurement method of a point spread function ofthe embodiment.

FIG. 10B illustrates a measurement method of a point spread function ofthe embodiment.

FIG. 11A illustrates an example of a captured image captured by an imagecapturing unit of the embodiment.

FIG. 11B illustrates an example of a captured image captured by an imagecapturing unit of the embodiment.

FIG. 11C illustrates an example of a captured image captured by an imagecapturing unit of the embodiment.

FIG. 12A illustrates a gap between white pixels included in an objectimage of the embodiment.

FIG. 12B illustrates a gap between white pixels included in an objectimage of the embodiment.

FIG. 12C illustrates a gap between white pixels included in an objectimage of the embodiment.

FIG. 13 illustrates an example of a captured image acquired by imagecapturing an object image displayed on a liquid crystal monitor of theembodiment.

FIG. 14 illustrates an example of the structure of an image processingdevice of the embodiment.

FIG. 15 illustrates an example of the system structure of a PSFmeasurement unit of the embodiment.

FIG. 16 illustrates an example of the functional structure of a PSFmeasurement processing program of the embodiment.

FIG. 17A illustrates an example of a pair of first and second testcharts of the embodiment.

FIG. 17B illustrates an example of a pair of first and second testcharts of the embodiment.

FIG. 18A illustrates another example of a pair of first and second testcharts of the embodiment.

FIG. 18B illustrates another example of a pair of first and second testcharts of the embodiment.

FIG. 19 illustrates an example of a first captured image acquired byimage capturing a first object image displayed on the liquid crystalmonitor of the embodiment.

FIG. 20 illustrates an example of a second captured image acquired byimage capturing a second object image displayed on the liquid crystalmonitor of the embodiment.

FIG. 21 is a flowchart of an example of a PSF measurement process of theembodiment.

DETAILED DESCRIPTION

In general, according to one embodiment, an image processing deviceincludes a hardware processor implemented by one or more processors. Thehardware processor acquires a first image of a first object and a secondimage of a second object which is different from the first object, theimages captured by an image capture processing device including an imagecapturing element and an optical system which images an object image onan image capturing surface of the image capturing element. The hardwareprocessor measures a point spread function of the optical system basedon the first image and the second image.

Embodiments will be described hereinafter with reference to theaccompanying drawings. The following description presents some examples,and the inventions are not limited by the description. Changes whichwould easily been conceived by a person having ordinary skill in the artare naturally encompassed by the scope of invention. For furtherclarification, in the figures, dimensions of each element may be changedfrom the actual embodiments and schematically illustrated. In somefigures, corresponding elements may be referred to by the same referencenumbers and explanation considered redundant will be omitted.

Now, an image capturing device which can acquire a distance to an objectfrom images captured by one camera (monocular camera) including anoptical system will be explained.

FIG. 1 illustrates an example of the structure of an image capturingdevice of an embodiment. Note that the image capturing device may bereferred to as ranging device. The image capturing device 1 captures animage and calculates a distance from a point of image capturing to anobject (may be referred to as a depth) using the captured image.

The image capturing device 1 includes an image capturing unit 11configured to capture an image and an image processing unit 12configured to process the captured image. The image capturing device 1may be realized as a device including the image capturing unit 11 andthe image processing unit 12, or may be a system including variousdevices such as an image capture processing device corresponding to theimage capturing unit 11 and an image processing device corresponding tothe image processing unit 12. The image capturing unit 11 has a functionto acquire an image of an object and data related to a distance to theobject at one image capturing. With this function, the image capturingunit 11 acquires, for example, an image in which the data of distance tothe object at the image capturing time are encoded (hereinafter referredto as distance image). Furthermore, the image processing unit 12 may berealized as an incorporated system installed in a computer or variouselectronic devices, for example.

The image capturing unit 11 includes, as in FIG. 1, a monocular cameraincluding a filter 21, lens 22, and image sensor 23. The filter 21includes a plurality of filter areas which pass light beams of differentwave length bands (color components). The filter 21 includes, forexample, a first filter area 211 and a second filter area 212 which arecolor filter areas of two different colors.

The image sensor 23 receives the light passing through the filter 21 andthe lens 22 and converts the received light into electric signals(light-to-electricity conversion). As the image sensor 23, for example,charge coupled device (CCD) or complementary metal oxide semiconductor(CMOS) will be used. The image sensor 23 includes at least two kinds ofimage capturing elements, and in this example, includes a first sensor231 including a red (R) light image capturing element, second sensor 232including a green (G) light image capturing element, and third sensor233 including a blue (B) light image capturing element. Each imagecapturing element receives light of corresponding wave length band andconverts the received light into electric signals. Through theanalogue/digital conversion of the electric signals, a color image canbe generated. In the following description, color component images of R,G, and B components of an image (or wavelength component images) may bereferred to as R image, G image, and B image. Note that, R, G, and Bimages may be generated using electric signals of image capturingelement of each of red, green and blue. That is, the image capturingunit 11 can generate at least one of a color image, R image, G image,and B image in one image capturing (one shot).

FIG. 2 illustrates an example of the structure of a system of the imageprocessing unit 12 of FIG. 1.

As in FIG. 2, the image processing unit 12 includes a CPU 31, RAM 32,nonvolatile memory 33, input/output unit 34, and communication unit 35,and includes a bus 36 which mutually connect the CPU 31, RAM 32,nonvolatile memory 33, input/output unit 34, and communication unit 35.

The CPU 31 is a hardware processor which controls operations of variouscomponents in the image processing unit 12. The CPU 31 may be a singleprocessor or may include a plurality of processors. The CPU 31 executesvarious programs loaded from the nonvolatile memory 33 to the RAM 32.The programs include an operating system (OS) and various applicationprograms. The application program includes an image processing program32A. The image processing program 32A includes a command group tocalculate a distance from a point of image capturing to an object usingat least one captured image of the object. Furthermore, the RAM 32 is astorage medium used as a main storage device. The nonvolatile memory 33is a storage medium used as an auxiliary storage device.

The input/output unit 34 is a module to execute inputs/outputs such asinput of an image from the image capturing unit 11, input ofinstructions from a user, and output of a display screen to a displayunit which is not shown. Instructions from a user may be input based onan operation to a keyboard, pointing device, operation button, and thelike. Or, if the display unit which is not shown is a touch screendisplay, instructions from a user may be input based on a touchoperation on the touch screen display.

The communication unit 35 is a device configured to execute wired orwireless communication. The communication unit 35 includes a transmitterwhich transmits signals and a receiver which receives signals. Thecommunication unit 35 executes communication to the external devicesthrough a network and communication with external devices aroundthereof. The external devices may include the image capturing unit 11(image capture processing device). That is, the communication unit 35may receive images from the image capturing unit 11.

Now, an example of the structure of the filter 21 of FIG. 1 withreference to FIG. 3.

The filter 21 has the filter areas that transmit light rays havingdifferent wavelength bands (color components), and two or more filterareas are point-asymmetric with respect to an optical center 213 of animage capturing device 1. The filter 21 includes, for example, filterareas of two colors: the first filter area 211 and the second filterarea 212. The center of the filter 21 corresponds to the optical center213 of the image capturing device 1 (lens 22). Each of the first filterarea 211 and the second filter area 212 has a shape point-asymmetricwith respect to the optical center 213. For example, the two filterareas 211 and 212 do not overlap, and the two filter areas 211 and 212constitute the entire area of the filter 21. In the example of FIG. 3,each of the first filter area 211 and the second filter area 212 has asemicircular shape formed by dividing the circular filter 21 by a linethrough the optical center 213.

For example, the first filter area 211 is a yellow (Y) filter area, andthe second filter area 212 is a cyan (C) filter area. Note that thefirst filter area 211 may be a magenta (M) filter area, and the secondfilter area 212 may be a yellow (Y) filter area. Furthermore, the firstfilter area 211 may be a cyan (C) filter area, and the second filterarea 212 may be a magenta (M) filter area.

The color filters transmit different wavelength bands. A part of awavelength band of light rays that passes one filter area may have, forexample, an overlap with a part of a wavelength band of light rays thatpasses another color filter area. A wavelength band of light rays thatpenetrates one color filter area may include, for example, a wavelengthband of light rays that passes another color filter area.

Note that each of the first filter area 211 and the second filter area212 may be a filter that changes transmissivity of any wavelength bands,or'a polarized filter that allows light rays polarized in any directionsto pass therethrough. Or, each filter area may be a microlens thatchanges light-condensing power of any wavelength bands. The filter thatchanges transmissivity of any wavelength bands may be, for example, aprimary color filter (RGB), a complementary color filter (CMY), a colorcorrection filter (CC-RGB/CMY), an infrared/ultraviolet cut filter, anND filter, or a masking shield. In a case where the first filter area211 and the second filter area 212 are microlenses, the lens 22 bringsabout biasing distribution of condensed light rays, which changes blurshapes.

Hereinafter, for easier understanding of explanation, mainly illustratedis a case where the first filter area 211 is a yellow (Y) filter areaand the second filter area 212 is a cyan (C) filter area in the filter21 illustrated in FIG. 3.

For example, the filter 21 in FIG. 3 being provided to an aperture ofthe camera configures a color aperture having a structure in which theaperture is divided into halves by two colors. Based on light rays thatpass the color aperture, the image sensor 23 generates an image. Thelens 22 may be disposed between the filter 21 and the image sensor 23 onan optical path of the light rays incident upon the image sensor 23. Thefilter 21 may be disposed between the lens 22 and the image sensor 23 onthe optical path of the light rays incident upon the image sensor 23. Ina case where lenses 22 are provided, the filter 21 may be disposedbetween two lenses 22.

Light rays having a wavelength band corresponding to the second sensor232 pass both the first filter area 211 of yellow color and the secondfilter area 212 of cyan color. Light rays having a wavelength bandcorresponding to the first sensor 231 pass the first filter area 211 ofyellow color and do not penetrate the second filter area 212 of cyancolor. Light rays having a wavelength band corresponding to the thirdsensor 233 pass the second filter area 212 of cyan color and do notpenetrate the first filter area 211 of yellow color.

Note that a state where light rays having a certain wavelength band passa filter or a filter area represents that the filter or the filter areatransmits the light rays having the wavelength band with hightransmissivity, and the state indicates that attenuation of the lightrays having the wavelength band due to the filter or the filter area (adecrease in light intensity) is extremely small. Furthermore, the statewhere light rays having a certain wavelength band do not pass a filteror a filter area represents that the light rays are shielded by thefilter or the filter area: for example, the filter or the filter areatransmits the light rays having the wavelength band with lowtransmissivity, and the state indicates that attenuation of the lightrays having the wavelength band due to the filter or the filter area isextremely large. For example, a filter or a filter area absorbs lightrays having a certain wavelength band so as to attenuate the light rays.

FIG. 4 illustrates exemplary transmissivity characteristics of the firstfilter area 211 and the second filter area 212. Although transmissivitywith respect to light rays having a wavelength band longer than 700 nmin wavelength bands of visible light is not illustrated, it should benoted that the transmissivity is close to transmissivity with respect toa wavelength band of 700 nm. In a transmissivity characteristic 214 ofthe first filter area 211 of yellow color illustrated in FIG. 4, lightrays having wavelength bands from about 620 nm to 750 nm correspondingto the R image and light rays having wavelength bands from about 495 nmto 570 nm corresponding to the G image are transmitted with hightransmissivity, and light rays having wavelength bands from about 450 nmto 495 nm corresponding to the B image are hardly transmitted. In atransmissivity characteristic 215 of the second filter area 212 of cyancolor, the light rays having the wavelength bands corresponding to the Bimage and the light rays having the wavelength bands corresponding tothe G image are transmitted with high transmissivity, and the light rayshaving the wavelength bands corresponding to the R image are hardlytransmitted.

Therefore, the light rays having the wavelength bands corresponding tothe R image (the first sensor 231) pass the first filter area 211 ofyellow color, and the light rays having the wavelength bandscorresponding to the B image (the third sensor 233) pass the secondfilter area 212 of cyan color. The light rays having the wavelengthbands corresponding to the G image (the second sensor 232) pass thefirst filter area 211 and the second filter area 212.

R and B images and blur shapes on the images change in accordance with adistance d to an object, specifically, in accordance with a differencebetween the distance d and a focusing distance d_(f). The focusingdistance d_(f) is a distance from an image-capturing position to afocused position where an image is not blurred (that is, a position infocus). The filter areas 211 and 212 have a shape point-asymmetric withrespect to the optical center 213 so that the blur shapes on the R and Bimages differ and shift depending on situations whether the object is onthe near side or on the deep side from the focusing distance d_(f).Directions of the shift in the blurs on the R and B images reversedepending on the situations whether the object is on the near side orthe deep side from the focusing distance d_(f) as seen from theimage-capturing position.

Now, changes in light rays by a color aperture the filter 21 and blurshapes will be explained with reference to FIG. 5.

In a case where an object 210 is on the deep side from the focusingdistance d_(f) (d>d_(f)), images captured by the image sensor 23 areblurred. Blur functions (point spread functions: PSF) indicating blurshapes on the images differ between the R, G, and B images. For example,a blur function 201R of the R image indicates a blur shape shifted tothe left, a blur function 201G of the G image indicates a balanced blurshape, and a blur function 201B of the B image indicates a blur shapeshifted to the right.

When the object 210 is at the focusing distance d_(f) (d=d_(f)), imagescaptured by the image sensor 23 are hardly blurred. Blur functionsindicating blur shapes on the images are substantially similar betweenthe R, G, and B images. In other words, a blur function 202R of the Rimage, a blur function 202G of the G image, and a blur function 202B ofthe B image indicate balanced blur shapes.

When the object 210 is on the near side from the focusing distance d_(f)(d<d_(f)), images captured by the image sensor 23 are blurred. Blurfunctions indicating blur shapes on the images differ between the R, G,and B images. In other words, a blur function 203R of the R imageindicates a blur shape shifted to the right, a blur function 203G of theG image indicates a balanced blur shape, and a blur function 203B of theB image indicates a blur shape shifted to the left.

In this manner, when the object 210 is on the near side or on the deepside from the focusing distance d_(f), the blur function 201R and theblur function 203R of the R image based on the light rays penetratingthe first filter area 211 of yellow color are asymmetric, and the blurfunction 201B and the blur function 203B of the B image based on thelight rays penetrating the second filter area 212 of cyan color are alsoasymmetric. The blur function 201R and the blur function 203R of the Rimage differ from the blur function 201B and the blur function 203B ofthe B image, respectively.

The image processing unit 12 (with the image processing program 32Atherein) of the image capturing device 1 uses the above-explainedcharacteristics and calculates a distance from an image-capturingposition to an object.

FIG. 6 illustrates an example of the structure of functions of the imageprocessing program 32A executed by the CPU 31 of the image processingunit 12. The image processing program 32A includes, for example, animage acquisition unit 41 and a distance calculation unit 42.

The image acquisition unit 41 acquires the G image in which the blurfunction indicates a balanced blur shape as a reference image.Furthermore, the image acquisition unit 41 acquires one or both of the Rand B images in which the blur functions indicate shifted blur shapes astarget images. The target image and the reference image are imagescaptured by one image capturing device at the same time.

The distance calculation unit 42 chooses a blur correction filter whichis added to the target image to increase a correlation with thereference image from a plurality of blur correction filters. The blurcorrection filters are functions to add different blurs to the targetimages. Now, a distance calculation process of the distance calculationunit 42 will be explained in detail.

The distance calculation unit 42 adds different blurs to the targetimages based on the target images and the reference image acquired andgenerates a corrected image in which the blur shapes of the targetimages are corrected. In this example, the distance calculation unit 42uses the blur correction filters preliminarily prepared on theassumption that a distance to an object is optional, generates acorrected image in which blur shapes of target images are corrected, andderives a distance by which a correlation between the corrected imageand the reference image becomes higher to calculate the distance to theobject.

The blur function of the captured image is determined based on a shapeof aperture of the image capturing device 1 and a distance between anobject position and a focusing position. FIG. 7 illustrates an exampleof the blur function of the reference image. As in FIG. 7, the shape ofaperture through which the wave length area corresponding to the secondsensor 232 is a symmetric circle, and the shape of blur indicated by theblur function does not change before and after the focus position, andthe width of blur changes based on a size of the distance between theobject and the focus position. The blur function indicating such a blurshape may be represented as a Gaussian function by which the blur widthis changed based on a size of the distance between the object positionand the focus position. Note that the blur function may be representedas a pillbox function by which the blur width is changed based on thedistance between the object position and the focus position.

FIG. 8 illustrates an example of the blur function of the target image.Note that, in each graph, center (x_(o), y_(o))=(0, 0). As in FIG. 8,the blur function of the target image (for example, R image) isrepresented as a one-side Gaussian function in which, if the object isin a farther side than the focus position: d>d_(f), the blur widthdecreases by the light attenuation in the first filter area 211 wherex>0. If the object is in a nearer side than the focus position: d<d_(f),the blur function is represented as a one-side Gaussian function inwhich the blur width decreases by the light attenuation in the firstfilter area 211 where x<0.

Furthermore, by analyzing the blur function of the reference image andthe blur function of the target image, a plurality of blur correctionfilters to correct the blur shape of the target image to the blur shapeof the reference image can be derived.

FIG. 9 illustrates an example of the blur correction filter. Note thatthe blur correction filter of FIG. 9 is the blur correction filter in acase where the filter 21 of FIG. 3 is used. As in FIG. 9, the blurcorrection filter passes the center point of the boundary line betweenthe first filter area 211 and the second filter area 212, anddistributed on the straight line (proximity of straight line) which isorthogonal to the boundary line. The distribution draws a mountain-likeshape as in FIG. 9 in which peak points (positions on the straight line,heights) and spread from the peak points differ per estimated distance.The blur shape of the target image can be corrected to various blurshapes at optional estimated distance using the blur correction filters.That is, correction images at optional estimated distances can begenerated.

The distance calculation unit 42 derives a distance where the blur shapeof the generated corrected image and the blur shape of the referenceimage are closest or matched from each pixel of the captured image. Thematching degree of the blur shapes can be derived by calculating thecorrelation between the corrected image and the reference image in arectangular area of optional size the center of which is each pixel. Thematching degree of the blur shape is calculated with a conventionalsimilarity evaluation method. The distance calculation unit 42 derivesthe distance by which the correlation between the corrected image andthe reference image becomes maximum to calculate the distance to theobject of each pixel.

For example, the conventional similarity evaluation method may be sum ofsquared difference (SSD), sum of absolute difference (SAD), normalizedcross-correlation (NCC), zero-mean normalized cross-correlation (ZNCC),and color alignment measure, for example.

As above, the distance calculation unit 42 generates a corrected imagein which a blur shape of a target image based on a filter area iscorrected by a blur correction filter with an estimated distance, andacquires a distance by which correlation between the generated correctedimage and a reference image is increased to derive a distance to anobject.

As can be understood from the above, the image capturing device 1 cancalculate a distance from an image-capturing position to an object basedon blurs on captured images. In the image capturing device 1, in orderto calculate the distance from the image-capturing position to theobject, a plurality of blur correction filters to correct blur functionsof target images to a blur function of a reference image arepreliminarily stored in, for example, a nonvolatile memory 33. The blurcorrection filters are preliminarily derived using a simplified pointspread function, and are preliminarily stored in the image capturingdevice 1. Specifically, a simplified point spread function ishypothetically set as a blur function derived from an image captured ina case where a distance to an object is a specific distance, and a blurcorrection filter is derived based on the specific distance. This isrepeated for certain times to derive blur correction filterscorresponding to various distances, and the blur correction filters arestored in the image capturing device 1.

However, the simplified point spread function is actually different fromthe blur function derived from the captured image, and thus, if adistance from an image-capturing position to an object is calculatedusing the blur correction filters derived based on the simplified pointspread function, there may be an error occurring in the calculateddistance. Thus, in order to improve the accuracy of acquisition of adistance to an object, what is required is a blur function actuallyderived from a captured image, that is, blur correction filters derivedusing a point spread function of an optical system of the camera insteadof the simplified point spread function. However, the measurement of apoint spread function of an optical system of the camera must beperformed in consideration of influences from the environmental lightand the display system, and thus, additional work steps are required.

For example, in a method of measuring a point spread function of anoptical system of the camera, a point light source is disposed in a darkroom, the point light source is image-captured, and a point spreadfunction of the point light source is measured. However, in this method,a masking process to place a black paper having a pinhole between thepoint light source and the camera, or the like must be performed, whichis an additional work step. Furthermore, the point spread functionchanges in accordance with a distance from the point light source to thecamera, and thus, various point light sources must be disposed tocapture images thereof. That is, moving of the point light sources mustbe performed at each time of image capturing, which is also anadditional word step.

Inventors of the present application would like to present a novelmethod of measuring a point spread function of an optical system of acamera using a liquid crystal monitor including a divided backlight inorder to solve the above-explained inconvenience.

Specifically, the inventors discovered a method in which a liquidcrystal monitor pixel pitch of which is 0.2 mm or less displays an imagein which only one pixel is white and the other pixels are black, and theimage is captured by a camera, and a point spread function of the whitepixel is measured. In this method, a size of one pixel is 0.2 mm×0.2 mmwhich is very small, and thus, one white pixel is regarded as a pointlight source, and the point spread function can be measured.

In the following description, the method will be further explained withreference to FIGS. 10A and 10B. In this method, as in FIG. 10A, a firstimage including one or more test charts TC₀ in each of which only onepixel is white (pixel value 255) and the other pixels are black (pixelvalue 0) is displayed in the liquid crystal monitor, and the first imageI₁ is captured by a camera from a position distant therefrom by a firstdistance. Thereby, a first captured image is obtained. Note that FIG.10A illustrates an example where the white pixels are arranged atregular intervals; however, the intervals between adjacent white pixelsmay be irregular. The intervals between adjacent white pixels will bedescribed later.

Then, as in FIG. 10B, a second image I₂ in which all pixels are black isdisplayed in the liquid crystal monitor, and the second image I₂ iscaptured by the camera from a position distant therefrom by the firstdistance. Thereby, a second captured image is obtained. Then, adifference between the first captured image and the second capturedimage is derived to measure a point spread function of the white pixelregarded as the point light source.

Through this method, through displaying the first image I₁ including alarge number of test charts TC₀ on the liquid crystal monitor, aplurality of point spread functions can be measured at once.Furthermore, by simply changing the first image I₁ to be displayed onthe liquid crystal monitor, display positions of the white pixelsregarded as the point light source can be changed, and thus, a work stepto move the point light sources (specifically, a work step to move thepoint light source in the horizontal direction orthogonal to thedirection of the distance between the point light sources and thecamera) can be reduced.

FIGS. 11A to 11C illustrate an example of a first captured imageobtained by displaying the first image I₁ including a large number oftest charts TC₀ and capturing the first image I₁ using the cameracorresponding to the image capturing unit 11 of FIG. 1. FIG. 11Aillustrates a first captured image in a case where the first image I₁(liquid crystal monitor) as an object is in a deep side from thefocusing distance d_(f) (d>d_(f)). FIG. 11B illustrates a first capturedimage in a case where the object is in the focusing distance d_(f)(d=d_(f)). FIG. 11C illustrates a first captured image in a case wherethe object is in a near side of the focusing distance d_(f) (d<d_(f)).

In the camera corresponding to the image capturing unit 11, light rayschange at the color aperture in which the circular filter 21 isdisposed, and thus, as in FIGS. 11A and 11C, the shifting of the blursare reversed based on whether the object is in the near side of thefocusing distance d_(f) or the object is in the deep side from thefocusing distance d_(f). Shapes of blurs of FIGS. 11A to 11C correspondto, point spread functions corresponding to a distance from the camerato each point light source (white pixel). Note that, for convenience ofexplanation, FIGS. 11A to 11C illustrate a first captured imageincluding a large number of blurs of same shape; however, in actualcases, shapes of blurs included in the first captured image differ inaccordance with distances from the camera.

As in FIGS. 11A to 11C, in a case where the first image I₁ including alarge number of test charts TC₀ is displayed on the liquid crystalmonitor, if intervals between the white pixels included in the testcharts TC₀ are too narrow, on the first captured image, a shape of blurcorresponding to a white pixel included in a test chart TC₀ may possiblyoverlap a shape of blur corresponding to a white pixel included inanother test chart TC₀ which is adjacent to the test chart TC₀. Such anoverlap may cause a failure of measurement of a point spread function.Thus, on the first image I₁, an interval d_(M) between adjacent whitepixels should preferably be greater than the following minimum valued_(Mmin).

The shape of blur (size of blur) on the first captured image changes inaccordance with a distance d from the camera to a white pixel as anobject. Specifically, as in FIGS. 11A to 11C, when the distance d fromthe camera to a white pixel approaches the focusing distance d_(f), thesize of the blur on the first captured image decreases, and when thedistance d departs from the focusing distance d_(f), the size of theblur on the first captured image increases.

The size of the blur on the first captured image can be formulated asthe following formula (1). In formula (1), b represents a size (radius)of the blur, a (=f/F, f: focusing distance, F: F value) represents anaperture diameter of the lens of the camera, p_(c) represents a pixelpitch of an image sensor of the camera, and d represents a distance fromthe camera to a white pixel.

$\begin{matrix}{b = {\frac{af}{2p_{c}}{{\frac{1}{d_{f}} - \frac{1}{d}}}}} & (1)\end{matrix}$

In order to prevent two blur shapes corresponding to two adjacent whitepixels from overlapping each other, as in FIG. 12A, an interval d_(c)between two adjacent white pixels should be greater than 2b on the firstcaptured image.

Now, as in FIG. 12B, an interval between two adjacent white pixels onthe first image I₁ is d_(M), and a pixel pitch of the liquid crystalmonitor on which the first image I₁ is p_(M), as in FIG. 12C, a similarrelation of the following formula (2) can be derived from the firstcaptured image, first image I₁, and various parameters of the camera.

f:d=d _(c) ×p _(c) :d _(M) ×p _(M)  (2)

Formula (2) can be rewritten into formula (3) as follows.

$\begin{matrix}{d_{c} = \frac{f \times d_{M} \times p_{M}}{u \times p_{c}}} & (3)\end{matrix}$

As described above, in order to prevent overlapping of two blur shapescorresponding to two adjacent white pixels, an interval d_(c) betweentwo adjacent white pixels on the first captured image is greater than2b, and thus, a minimum value d_(Mmin) of the interval between twoadjacent white pixels on the first image I₁ is, based on formulae (1)and (3), derived as formula (4) as follows.

$\begin{matrix}{\frac{f \times d_{Mmin} \times p_{M}}{u \times p_{c}} = {{2b} = {\left. {\frac{af}{p_{c}}{{\frac{1}{d_{f}} - \frac{1}{d}}}}\Leftrightarrow d_{Mmin} \right. = {\frac{a}{p_{M}}{\frac{d - d_{f}}{d_{f}}}}}}} & (4)\end{matrix}$

As explained above, the first image I₁ including a large number of testcharts TC₀ arranged such that an interval between two adjacent whitepixels is greater than the above d_(Mmin) can be displayed, and thus, aplurality of point spread functions can be measured at once.

However, it was revealed that the above measurement method cannot removean influence of backlight of the liquid crystal monitor (in other words,influence of a display system), and a true and accurate point spreadfunction is not measurable, and thus, it is inconvenient.

In the following description, this inconvenience will be described. Thefirst image I₁ of FIG. 10A, that is, the first image I₁ including onewhite pixel in an optional area x is represented as g₁(x), an influenceof the backlight of the liquid crystal monitor lit to display the onewhite pixel is represented as h(x), and influence of an environment atthe time of image capturing of the first image I₁ (hereinafter referredto as ambient image influence) is represented as k(x). In that case, thefirst captured image will be represented as formula (5) as follows.

First captured image=f*g ₁(x)+h(x)+k(x)  (5)

As indicated in formula (5), the first captured image is generated byconvolution of the point spread function corresponding to the firstdistance with respect to the first image I₁, wherein influence by abacklight and ambient image influence are added to a result ofconvolution (in other words, influence by light emitted from an object).FIG. 13 schematically illustrates a first captured image CI₀, wherein awhite part represents a white pixel PX_(W), a black part represents ablack pixel PX_(B), and a hatched part represents an influence h(x) ofthe backlight.

The second capture image is obtained by image capturing a second imageI₂ in which all pixels are black as in FIG. 10B. The second image I₂does not include a white pixel which is regarded as a point lightsource, and thus, an influence by light emitted from the object is notadded to the second captured image. Furthermore, in general, when onlythe black pixels are displayed, the backlight of the liquid crystalmonitor is turned off (attenuated), and thus, an influence by thebacklight of the liquid crystal monitor is not added to the secondcaptured image. That is, only the ambient image influence is added tothe second captured image. If the environment at the time ofimage-capturing of the second image I₂ is the same as the environment atthe time of image-capturing of the first image I₁, a difference betweenthe first captured image and the second captured image will berepresented as formula (6) as follows.

$\begin{matrix}\begin{matrix}{{{First}\mspace{14mu} {captured}\mspace{14mu} {image}\text{-}{second}\mspace{14mu} {captured}\mspace{14mu} {image}} =} & {\left\{ {{f*{g_{1}(x)}} +} \right.} \\ & {\left. {{h(x)} + {k(x)}} \right\} -} \\ & {{k(x)}} \\{=} & {{{f*{g_{1}(x)}} +}} \\ & {{h(x)}} \\{=} & {{{f(x)} + {h(x)}}}\end{matrix} & (6)\end{matrix}$

As can be understood from formula (6), in the method of measuring apoint spread function using a liquid crystal monitor including a dividedbacklight, there is an inconvenience in which the influence h(x) by thebacklight of the liquid crystal monitor (corresponding to the hatchedpart of FIG. 13). In the following description, a measurement method ofthe point spread function which can solve the inconvenience will beexplained.

FIG. 14 illustrates an example of the structure of an image processingdevice of the present embodiment. The image processing device 50includes an image capturing unit 51 configured to capture an image, anda PSF measurement unit 52 configured to measure a point spread functionby processing the captured image. The image processing device 50 may berealized as one device including both the image capturing unit 51 andthe PSF measurement unit 52, or may be realized as a system including animage capture processing device corresponding to the image capturingunit 51 and a PSF measurement processing device corresponding to the PSFmeasurement unit 52. Furthermore, the image processing device 50 may beintegrated in the image capturing device 1 of FIG. 1. In that case,since the image capturing unit 51 corresponds to the image capturingunit 11 of FIG. 1, only the PSF measurement unit 52 may be incorporatedin the image capturing device 1.

The image capturing unit 51 includes a filter, lens, and image sensor,wherein the image sensor includes at least one type of image capturingelement and an optical system which images an object image on an imagecapturing surface of the image capturing element. The PSF measurementunit 52 has a function to measure a point spread function correspondingto a certain distance in the optical system of the image capturing unit51 based on the images captured by the image capturing unit 51. The PSFmeasurement unit 52 may be realized as an incorporated system stored ina computer or various electronic devices (for example, image capturingdevice 1).

FIG. 15 illustrates an example of the structure of a system of the PSFmeasurement unit 52 of FIG. 14.

As in FIG. 15, the PSF measurement unit 52 includes a CPU 61, RAM 62,nonvolatile memory 63, input/output unit 64, and communication unit 65,and includes a bus 66 which mutually connect the CPU 61, RAM 62,nonvolatile memory 63, input/output unit 64, and communication unit 65.

The CPU 61 is a hardware processor which controls operations of variouscomponents in the PSF measurement unit 52. The CPU 61 may be a singleprocessor or may include a plurality of processors. The CPU 61 executesvarious programs loaded from the nonvolatile memory 63 to the RAM 62.The programs include an operating system (OS) and various applicationprograms. The application program includes a PSF measurement processprogram 62A. The PSF measurement process program includes a commandgroup to measure a point spread function corresponding to a certaindistance in the optical system of the image capturing unit 51 using theimages captured by the image capturing unit 51. Furthermore, the RAM 62is a storage medium used as a main storage device. The nonvolatilememory 63 is a storage medium used as an auxiliary storage device. Inthe nonvolatile memory 63, for example, object images of various kindsare stored.

The input/output unit 64 is a module to execute inputs/outputs such asinput of an image from the image capturing unit 51, input ofinstructions from a user, and output of a display screen to a displayunit which is not shown. Instructions from a user may be input based onan operation to a keyboard, pointing device, operation button, and thelike. Or, if the display unit which is not shown is a touch screendisplay, instructions from a user may be input based on a touchoperation on the touch screen display.

The communication unit 65 is a device configured to execute wired orwireless communication. The communication unit 65 includes a transmitterwhich transmits signals and a receiver which receives signals. Thecommunication unit 65 executes communication to the external devicesthrough a network and communication with external devices aroundthereof. The external devices may include the image capturing unit 51(image capture processing device). That is, the communication unit 65may receive images from the image capturing unit 51.

FIG. 16 illustrates an example of the structure of functions of PSFmeasurement process program 62A executed by the CPU 61 of the PSFmeasurement unit 52. The PSF measurement process program 62A includes,for example, an image display instruction unit 71, image acquisitionunit 72, PSF calculation unit 73, and PSF output unit 74.

The image display instruction unit 71 has a function to instruct theliquid crystal monitor to display various kinds of object images. Thatis, the image display instruction unit 71 sequentially reads variouskinds of object images which are preliminarily stored in the nonvolatilememory 63, and outputs the read object images and a display command todisplay the object images to the liquid crystal monitor through thecommunication unit 65.

Note that, in the nonvolatile memory 63, at least two kinds of objectimages, that is, a first object image and a second object image arepreliminarily stored.

Now, the first object image and the second object image will beexplained. Both the first and second object images include one or moretest charts including one or more white pixels and a plurality of blackpixels. The first object image includes one or more first test charts,and the second object image includes one or more second test charts. Thefirst test charts include n+1 white pixels and a plurality of blackpixels. The second test charts includes n white pixels and a pluralityof black pixels. That is, the first and second test charts have arelationship in which a difference of white pixels therebetween is 1.

A specific example of a pair of the first and second test charts will beexplained with reference to FIGS. 17A and 17B.

FIGS. 17A and 17B illustrate a test chart in which n is 1, and therein,FIG. 17A illustrates an example of the first test chart and FIG. 17Billustrates an example of the second test chart corresponding to FIG.17A. The first test chart TC₁ of FIG. 17A includes a rectangularstructure of 5×4 and includes two white pixels and 18 black pixels. Thesecond test chart TC₂ of FIG. 17B includes, as with the first test chartTC₁, a rectangular structure of 5×4 and includes one white pixel and 19black pixels. The first and second test charts TC₁ and TC₂ of FIGS. 17Aand 17B correspond to a pair of first and second test charts in whichthe number of white pixels is minimum.

Furthermore, another specific example of a pair of the first and secondtest charts will be explained with reference to FIGS. 18A and 18B.

FIGS. 18A and 18B illustrate a test chart in which n is 4, and therein,FIG. 18A illustrates an example of the first test chart and FIG. 18Billustrates an example of the second test chart corresponding to FIG.18A. The first test chart TC₁ of FIG. 18A includes a square structure of5×5 and includes five cross-shaped white pixels and 20 black pixelsaround the cross-shaped white pixel. The second test chart TC₂ of FIG.18B includes, as with the first test chart TC₁, a square structure of5×5 and includes four cross-like shaped white pixels which is similar tothat of FIG. 18A but one white pixel is replaced with one black pixel.The first and second test charts TC₁ and TC₂ of

FIGS. 18A and 18B include more white pixels as compared to the first andsecond test charts TC₁ and TC₂ of FIGS. 17A and 17B. Thus, inconsideration of an exposure time of the image capturing unit 51(camera), image capturing in a short time can be performed.

Note that, in this example, as pairs of the first and second test chartsTC₁ and TC₂, the structures of FIGS. 17A, 17B, 18A, and 18B areexemplified; however, the first and second test charts TC₁ and TC₂ maybe structured optionally as long as a relationship that both the firstand second test charts TC₁ and TC₂ include one or more white pixels anda difference between white pixels of the first and second test chartsTC₁ and TC₂ is kept. Furthermore, several pairs of the first and secondtest charts TC₁ and TC₂ are prepared, and the pairs of the first andsecond test charts TC₁ and TC₂ may be switched to a suitable pair inaccordance with the distance from the image capturing unit 51.

The image acquisition unit 72 instructs, when the display command isoutput with the first object image to the liquid crystal monitor fromthe image display instruction unit 71, the image capturing unit 51 toimage-capture the first object image displayed on the liquid crystalmonitor (outputs an image capturing command). The instruction may be anotification to prompt a user to capture the first object image. Notethat the image display instruction unit 71 may have the function. Theimage acquisition unit 72 acquires the first captured image indicativeof the first object image captured by the image capturing unit 51.

The image acquisition unit 72 has a function to instruct, when thedisplay command is output with the second object image to the liquidcrystal monitor from the image display instruction unit 71, the imagecapturing unit 51 to image-capture the second object image displayed inthe liquid crystal monitor. The instruction may be a notification toprompt a user to capture the second object image. Note that the imagedisplay instruction unit 71 may have the function. The image acquisitionunit 72 acquires the second captured image indicative of the secondobject image captured by the image capturing unit 51.

The first and second object images are captured in the same environment.That is, the first and second object images are captured in theenvironment where the distance from the image capturing unit 51 andambient image are the same.

The PSF calculation unit 73 has a function to calculate a point spreadfunction of one white pixel which in included in the first test chartTC₁ but not included in the second test chart TC₂ by acquiring adifference between the first and second captured images. Thereby, a trueand accurate point spread function corresponding to a certain distanceof the optical system of the image capturing unit 51 can be calculated.

Now, a mechanism to calculate a true and accurate point spread functionwill be explained.

In the following description, an object image including the first testchart TC₁, that is, the first object image including n+1 white pixels inan optional area x will be represented as g_(n+1)(x), and influence ofthe backlight of the liquid crystal monitor lit to display the n+1 whitepixels included in the first test chart TC₁ will be represented ash_(n+1)(x). Furthermore, when ambient image influence at the time ofimage-capturing the first object image is represented as k(x), the firstcaptured image will be represented as formula (7) as follows.

First captured image=f*g _(n+1)(x)+h _(n+1)(x)+k(x)  (7)

As indicated in formula (7), the first captured image is generated byconvolution of the point spread function corresponding to a distancebetween the image capturing unit 51 and the first object image (liquidcrystal monitor) to the first object image, wherein influence by abacklight and ambient image influence are added to a result ofconvolution. FIG. 19 schematically illustrates a first captured imageCI₁, wherein a white part represents a white pixel PX_(W), a black partrepresents a black pixel PX_(B), and a hatched part represents aninfluence h_(n+1)(x) of the backlight.

Similarly, the second object image including the second test chart TC₂,that is, the second object image including n white pixels in an optionalarea x will be represented as g_(n)(x), and influence of the backlightof the liquid crystal monitor lit to display the n white pixels includedin the second test chart TC₂ will be represented as h_(n)(x).Furthermore, when ambient image influence at the time of image-capturingthe second object image is the same as the environment at the time ofimage-capturing the first object image, and thus, the point spreadfunction and the ambient image influence convoluted in the second objectimage become formula (7) above. The second captured image will berepresented as formula (8) as follows.

Second captured image=f*g _(n)(x)+h _(n)(x)+k(x)  (8)

FIG. 20 schematically illustrates a second captured image CI₂, wherein awhite part represents a white pixel PX_(W), a black part represents ablack pixel PX_(B), and a hatched part represents an influence h_(n)(x)of the backlight. Note that FIGS. 19 and 20 illustrate examples of firstand second captured images CI₁ and CI₂ captured when the first andsecond object images are in the focusing distance d_(f). If the firstand second object images are in a near side of or in a deep side fromthe focusing distance d_(f), each area occupied by each of the partsPX_(W), PX_(B), and h(x) included in the first and second capturedimages CI₁ and CI₂ changes greatly when departing from the focusingdistance d_(f).

Based on formulae (7) and (8), a difference between the first and secondcaptured images will be represented as formula (9) as follows.

$\begin{matrix}\begin{matrix}{{{First}\mspace{14mu} {captured}\mspace{14mu} {image}\text{-}{second}\mspace{14mu} {captured}\mspace{14mu} {image}} =} & {\left\{ {{f*{g_{n + 1}(x)}} +} \right.} \\ & {\left. {{h_{n + 1}(x)} + {k(x)}} \right\} -} \\ & {{\left\{ {f*g_{n}x} \right) +}} \\ & \left. {{h_{n}(x)} + {k(x)}} \right\} \\{=} & {\left\{ {{f*{g_{n + 1}(x)}} +} \right.} \\ & {\left. {h_{n + 1}(x)} \right\} -} \\{=} & \left. {\left\{ {f*g_{n}x} \right) + {h_{n}(x)}} \right\}\end{matrix} & (9)\end{matrix}$

Since the first and second test charts TC₁ and TC₂ include one or morewhite pixels, and thus, the backlight is lit when either test chart isdisplayed, and since a difference between the white pixels included inthe first and second test charts TC₁ and TC₂ is 1, the influence of thebacklight lit to display n+1 white pixels and the influence of thebacklight lit to n white pixels can be regarded as substantially thesame. Thus, formula (9) can be rewritten into formula (10) as follows.

$\begin{matrix}\begin{matrix}{\left. {\left\{ {{f*{g_{n + 1}(x)}} + {h_{n + 1}(x)}} \right\} - \left\{ {f*g_{n}x} \right) + {h_{n}(x)}} \right\}  =} & {{f*\left\{ {{g_{n + 1}(x)} -} \right.}} \\ & \left. {g_{n}(x)} \right\} \\{=} & {{f*{g_{1}(x)}}} \\{=} & {{f(x)}}\end{matrix} & (10)\end{matrix}$

That is, as in formulae (9) and (10), by deriving a difference betweenthe first captured image obtained when the first object image includingthe first test chart TC₁ is captured and the second captured imageobtained when the second object image including the second test chartTC₂ is captured, the PSF calculation unit 73 can calculate a true andaccurate point spread function corresponding to a certain distance ofone white pixel which is included in the first test chart TC₁ but notincluded in the second test chart TC₂.

The PSF output unit 74 outputs PSF information indicative of the pointspread function calculated by the PSF calculation unit 73 to variouselectronic devices (for example, image processing unit 12). Thus, whenthe PSF measurement unit 52 is incorporated in the image capturingdevice 1, the PSF measurement unit 52 can measure the point spreadfunction of the optical system of the image capturing nit 11, and thus,the image capturing device 1 can generate blur correction filters basedon the PSF information indicative of the point spread function of theoptical system of the image capturing unit 11. That is, since the blurcorrection filters based on a measured value can be preliminarilyprepared, accuracy of acquisition of a distance by the image capturingdevice 1 can be improved.

Note that the PSF information may indicate the measured value of thepoint spread function calculated by the PSF calculation unit 73, or mayindicate, for example, a simplified measured value of the point spreadfunction calculated by the PSF calculation unit 73 based on Gaussianfunction or Zernike polynomials.

FIG. 21 is a flowchart of an example of the PSF measurement processexecuted by the PSF measurement unit 52.

Initially, the image display instruction unit 71 reads a first objectimage to be stored in the nonvolatile memory 63 and outputs the firstobject image together with a display commands to the liquid crystalmonitor (step S1). Through this step, the first object image isdisplayed on the liquid crystal monitor.

Then, the image display instruction unit 71 instructs the imagecapturing unit 51 to capture the first object image displayed on theliquid crystal monitor. When the first object image is captured based onthe instruction, the image acquisition unit 72 acquires a first capturedimage CI₁ indicative of the first object image (step S2).

Then, the image display instruction unit 71 reads a second object imagestored in the nonvolatile memory 63 and outputs the second object imagetogether with a display command to the liquid crystal monitor (step S3).Through this step, the second object image is displayed on the liquidcrystal monitor.

Then, the image display instruction unit 71 instructs the imagecapturing unit 51 to capture the second object image displayed on theliquid crystal monitor. When the second object image is captured basedon the instruction, the image acquisition unit 72 acquires a secondcaptured image CI₂ indicative of the second object image (step S4).

Then, the PSF calculation unit 73 calculates each point spread functioncorresponding to a distance to each white pixel which is a measurementtarget based on the first and second captured images CI₁ and CI₂acquired by the image acquisition unit 72 (step S5).

Then, the PSF output unit 74 outputs PSF information indicative of eachcalculated point spread function to various electronic devices (stepS6), and ends the PSF measurement process.

Note that the image processing device 50 may further include a movementmechanism which can move to a direction of a distance between an objectand the image capturing unit 51 and a control unit configured to controlthe movement mechanism. Thereby, the distance d between the imagecapturing unit 51 and the object can easily be adjusted.

With the above-described embodiment, an image processing device 50includes an image capturing unit 51 including an optical system and aPSF measurement unit 52. The PSF measurement unit 52 sequentiallydisplays a first object image including a first test chart TC₁ and asecond object image including a second test chart TC₂ on a liquidcrystal monitor, and based on a first captured image CI₁ obtained by theimage capturing unit 51 capturing the first object image, and a secondcaptured image CI₂ obtained by the image capturing unit 51 capturing thesecond object image, a point spread function of the optical system ofthe image capturing unit 51 is measured.

Thus, measurement of the point spread function of the optical system ofthe camera (image capturing unit 51) can easily be executed.

Furthermore, since the measurement of the point spread function of theoptical system of the camera can easily be executed, subsidiary effectsthat evaluation of optical performance of lens of camera and correctionof lens aberration can easily be performed can be achieved.

Specifically, since the measurement of the point spread function of theoptical system of the camera can easily be executed, as evaluation ofthe optical performance of the lens of the camera, imaging performancecan easily be evaluated in a focusing position of the camera, and blurdeterioration can easily be evaluated in the positions other than thefocusing position of the camera.

Furthermore, since the measurement of the point spread function of theoptical system of the camera can easily be executed, for example, singlecolor aberration and color aberration can easily be corrected, and thus,an image deteriorated by the lens aberration can be restored to itsoriginal quality.

Furthermore, since the PSF measurement unit 52 can be incorporated inthe image capturing device 1, the measurement of the point spreadfunction of the optical system of the image capturing nit 11, andtherein, a blur correction filter can be derived based on themeasurement result, and thus, acquisition of a distance from theimage-capturing position to the object by the image processing unit 12can be performed with higher accuracy.

While certain embodiments have been described, these embodiments havebeen presented by way of example only, and are not intended to limit thescope of the inventions. Indeed, the novel embodiments described hereinmay be embodied in a variety of other forms; furthermore, variousomissions, substitutions and changes in the form of the embodimentsdescribed herein may be made without departing from the spirit of theinventions. The accompanying claims and their equivalents are intendedto cover such forms or modifications as would fall within the scope andspirit of the inventions.

What is claimed is:
 1. An image processing device, comprising: ahardware processor implemented by one or more processors; wherein thehardware processor which: acquires a first image of a first object and asecond image of a second object which is different from the firstobject, the images captured by an image capture processing deviceincluding an image capturing element and an optical system which imagesan object image on an image capturing surface of the image capturingelement; and measures a point spread function of the optical systembased on the first image and the second image.
 2. The image processingdevice of claim 1, wherein both the first object and the second objectare images displayed on a liquid crystal monitor including a dividedbacklight.
 3. The image processing device of claim 1, wherein both thefirst object and the second object are images including at least onewhite pixel, and the first object includes one more white pixel than thesecond object does.
 4. The image processing device of claim 1, whereinthe first object includes two adjacent white pixels, and the secondobject includes the two adjacent white pixels one of which is replacedwith a black pixel.
 5. The image processing device of claim 1, whereinthe first object includes a plurality of white pixels adjacent to eachother in a cross shape, and the second object includes the white pixelsadjacent to each other in a cross shape, center of which is replacedwith a black pixel.
 6. The image processing device of claim 1, whereinthe hardware processor measures the point spread function of the opticalsystem based on a difference between the first image and the secondimage.
 7. The image processing device of claim 1, wherein the hardwareprocessor acquires a first color image indicative of a shape in which ablur function is not biased and a second color image indicative of ashape in which the blur function is biased, where the first color imageand the second color image are obtained by further image capturing of athird object by the image capture processing device.
 8. The imageprocessing device of claim 7, wherein the hardware processor addsdifferent blurs based on the measured point spread function to thesecond color image, generates a plurality of correction images, andcalculates a distance to the third object based on a correlation betweenthe first color image and correction images.
 9. An image capturingdevice comprising: an image capture unit including an image capturingelement and an optical system which images an object image on an imagecapturing surface of the image capturing element, the image capture unitwhich captures a first image of a first object and a second image of asecond object which is different from the first object; and ameasurement unit which measures a point spread function of the opticalsystem based on the first image and the second image.
 10. An imageprocessing method comprising: acquiring a first image of a first objectand a second image of a second object which is different from the firstobject, the images captured by an image capture processing deviceincluding an image capturing element and an optical system which imagesan object image on an image capturing surface of the image capturingelement; and measuring a point spread function of the optical systembased on the first image and the second image.
 11. The image processingmethod of claim 10, wherein both the first object and the second objectare images displayed on a liquid crystal monitor including a dividedbacklight.
 12. The image processing method of claim 10, wherein both thefirst object and the second object are images including at least onewhite pixel, and the first object includes one more white pixel than thesecond object does.
 13. The image processing method of claim 10, whereinthe first object includes two adjacent white pixels, and the secondobject includes the two adjacent white pixels one of which is replacedwith a black pixel.
 14. The image processing method of claim 10, whereinthe first object includes a plurality of white pixels adjacent to eachother in a cross shape, and the second object includes the white pixelsadjacent to each other in a cross shape, center of which is replacedwith a black pixel.
 15. The image processing method of claim 10, furthercomprising measuring the point spread function of the optical systembased on a difference between the first image and the second image. 16.The image processing method of claim 10, further comprising acquiring afirst color image indicative of a shape in which a blur function is notbiased and a second color image indicative of a shape in which the blurfunction is biased, where the first color image and the second colorimage are obtained by further image capturing of a third object by theimage capture processing device.
 17. The image processing method ofclaim 16, further comprising: adding different blurs based on themeasured point spread function to the second color image, generating aplurality of correction images, and calculating a distance to the thirdobject based on a correlation between the first color image andcorrection images.